In augmented reality applications the real world is enhanced with virtual computer graphics. Modern computing hardware and software enables the creation of rich, realistic 3D virtual environments. Such 3D virtual environments are widely used for gaming, education/training, prototyping, and any other application where a realistic virtual representation of the real world is useful. To enhance the realism of these 3D virtual environments, physics simulations are used to control the behavior of virtual objects in a way that resembles how such objects would behave in the real world under the influence of Newtonian forces. This enables their behavior to be predictable and familiar to a user.
It is, however, difficult to enable a user to interact with these 3D virtual environments in a natural manner where direct-as-possible interactions with virtual objects are enabled. Most interactions with 3D virtual environments happen via indirect input devices such as mice, keyboards or joysticks. Other, more direct input paradigms have been explored as means to manipulate virtual objects in such virtual environments. Among them is pen-based input control, and also input from vision-based multi-touch interactive surfaces. However, in such instances there is the mismatch of input and output. Pen-based and multi-touch input data is inherently 2D which makes many interactions with the 3D virtual environments difficult if not impossible. For example, the grasping of objects to lift them or to put objects into containers etc. cannot be readily performed using 2D inputs.
In the real world users readily apply a variety of strategies to manipulate objects such as pushing, pulling, dragging and grasping using the full dexterity of human hands. Previous approaches to enabling virtual objects to be manipulated in these ways have involved detecting and tracking iconic gestures made by human hands and using those detected and tracked gestures to put virtual objects under grasping and other types of control. Other approaches have involved using inverse kinematics to track an articulated model of a human hand and using the tracked model to enable manipulation of objects in some ways.
The embodiments described below are not limited to implementations which solve any or all of the disadvantages of known augmented reality systems.